In conventional image processing in which digital signal processing is applied to input image data, the input image data is received from a sensor input such as a camera and a defective pixel correction process is performed on the entire input image data. In an image processor, intermediate image data obtained as output image data from the execution of the defective image correction process is input into a subsequent process, for example a shading correction process, and output image data from the execution of the shading correction process is further input into a subsequent process, and so on. In this way, various processes may be applied to image data, which is intermediate data between processes.
In the case where each process task such as defective pixel correction and shading correction process tasks receives input image data and outputs image data having the same data size as the input image data and a subsequent process receives the resulting output data and applies processing to the data as described above, the process tasks can be straightforwardly programmed, developed and modified by writing a program so as to use a memory area (frame memory) allocated for storing data of the size of input image data.
However, when data of the size of input image data is passed from one process task to another, a frame memory capable of holding data of the size of the input image data is required. Specifically, if the size of input image data is 20 Mbytes, 20 Mbytes of frame memory is required for holding the input image data and another 20 Mbytes of frame memory is required for holding intermediate image data. A total of 40 Mbytes of memory area is required for the entire process.
Therefore, when such processes are built into an embedded device or the like, typically the fact that each process task refers to a limited range of input data to calculate data of a certain pixel location is used to modify the process tasks so that the process tasks are executed in a pipeline, thereby minimizing the amount of data retained between the processes.
For example, an image processing apparatus has been proposed that has multiple image processing means that performs line processing by pipeline control (for example see Japanese Patent Application Laid-Open Publication No. 2005-11380).
For example, suppose that process tasks are to be modified so that the tasks are executed in a pipeline and that a process task, for example a process task that performs defective pixel correction, reads data in a rectangular region with vertical coordinates of v−1 to v+1 and horizontal coordinates of h−1 to h+1 in input image data in order to calculate data to be written in locations with vertical coordinates v and horizontal coordinates h in intermediate image data, performs an operation on the read data, and then writes output data. Here, let Vin denote the vertical coordinate of input image data written at a particular timing, Hin denote the horizontal coordinate of the input image data written in that timing, Va denote the vertical coordinate position in an image of intermediate image data to be calculated by the defective pixel correction process task, and Ha denote the horizontal coordinate position in the image. When the process is modified so as to be executed in a pipeline, a difference of −1 in relative vertical coordinate and a difference of −1 relative horizontal coordinate need to be provided between a pair of Vin and Hin and a pair of Va and Ha according to a range of data to be read by the defective pixel correction process task. That is, a delay needs to be provided in the process. The difference between relative coordinate positions will be referred to as amount of processing delay. Each time input image data at the coordinates Vin and Hin is written, Vin and Hin are incremented in raster scan order, an operation of the defective pixel correction process task is performed on data in the position Va, Ha, and Va and Ha are also incremented in the raster scan order.
When the cycle of writing input image data and executing the process task is repeated on a pixel-by-pixel basis while each pixel location is incremented in the raster scan order, an appropriate amount of process delay need to be provided for each process task.
Here, the raster scan order process is a process in which the horizontal coordinate is incremented to advance the scanning motion with respect to the upper-left of input image data and, after the horizontal coordinate is incremented to the rightmost end of the input image data, the vertical coordinate is incremented by 1 and the horizontal coordinate is reset to 0, thereby increasing the vertical and horizontal coordinates.
In order for the defective pixel correction process task to perform processing on certain vertical and horizontal coordinates Vaα, Haα in a certain execution cycle α in the pipeline execution as described above, rectangular region data (Vaα−1, Haα−1) . . . (Vaα+1, Haα+1) of input image is required. The rectangular region data is equivalent to a rectangular region (Vinα−2, Hinα−2) . . . (Vinα, Hinα).
In a next execution cycle α+1, the defective pixel correction process task requires rectangular region data (Vaα−1, Haα) . . . (Vaα+1, Haα+2) of the input image. The rectangular region data to be accessed in the execution cycle α+1 is equivalent to a rectangular region (Vinα−2, Hinα−1) . . . (Vinα, Hinα+1).
Therefore, data in the location (Vinα−2, Hinα−2) of the input image is accessed in the execution cycle α whereas the data in the location (Vinα−2, Hinα−2) is not required in the execution cycle α+1 and data in the location (Vinα, Hinα+1) is newly required instead.
Accordingly, in a certain execution cycle γ in the pipeline execution, an area to store data in a rectangular region from the starting point (Vaγ−1, Haγ−1) to the coordinates (Vaγ+1, Haγ+1) in raster scan order on the input image, that is, data in the region (Vinγ−2, Hinγ−2) . . . (Vinγ, Hinγ) of input image, that is, 2 lines+3 pixels of data, needs to be allocated on memory so that the defective pixel correction task can be continuously executed. Pipeline execution as described above can significantly reduce the amount of memory required for executing processing, as compared with execution in which a memory area for one frame of input image data is allocated and date is passed from one process to another.
A memory area that holds only a number of lines+a number of pixels of data being passed from one process task to another executed in a pipeline in this way is hereinafter referred to as line memory.
However, in order to use pipeline execution to reduce the required amount of memory, a user needs to write process tasks so as to use line memories. The user need to consider data access relationship between the process tasks to add an appropriate processing delay amount to each process, and to write the process tasks in a source file by taking into consideration the required line memory size. Productivity had been reduced by this procedure.
Furthermore, when the user has made a modification to a process task, the amount of processing delay and the required line memory size need to be changed according to the modification. Recalculation of the amount of processing delay and the required line memory size for the modification further decreases the productivity. If the user miscalculates the amount of processing delay and the required line memory size, it is difficult to identify errors. These problems will be especially remarkable if processes to be implemented are complicated, that is, data is passed from one process to another in a complicated manner.
As has been described above, writing a process task so as to use line memory has a problem that it decreases productivity—compared with writing a process task so as to use frame memory.